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*Based on Greenfield & Gindorf, “Risk as a Resource - A New Paradigm”, 1996 


What can be done to balance the wide JPL 
variety of risks a project needs to manage? 
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* Note: Resources include some areas which may not be applicable to software 
(i.e. mass & power) 


Software Quality Assurance JPL 
/ V&V Program Content 
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piloted only techniques were not included in the program 
content area 

Content ranged from a super minimal approach to a full up 
QA / V&V program 


Software Quality Assurance JPL 
/ V&V Program Content (continued) 
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There are no 100% certain, 0% Risk programs! 
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what it “is not” 
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Risk Balance Profile 

Software Quality and V&V Program Guide 
“FODORS” 


Performance 


Coats 


Schedule 


COST RISK 
FACTORS 


SCHEDULE 

RISK 

FACTORS 


Program Title 


Software 


Program 


Contents 


Schedule Pressure Resolved 

by S 

Repeat Testing 
Changing Requirements 
S/W Faults Could Impact 
System Testi 


Late Problem Identification 
Repair and Repeat Testing 
Changing Requirements 
S/W Faults Could Impact 
System Testing 


Very High Risk 
Minimal QA/V&V P 


Program Content 


Testing 

< T1 -Accept Test (pass/fai l 
w/o metrics) 

< T2-Functional Test 
(pass/fail) 


A1 -Hazards Analysis 
A2-S/W FMEA (if 
applicable for critical 
functions only) 


Based on Trade-offs of 


Based on Trade-offs of Ri 


353BE3F5! 


Mitigation, Content determined by user 


on. Content determined by user 


< 

None 

Other 


( 

None 

Residual Risks 

< 

( 

< 

RI - Lack of confidence in 
acceptability of S/W to meet 
system’s needs-Tl 
R2 - Unknown functional 
and system margins-T2 
R3 - Inconsistent S/W 


requirements with respect to 
the system’s functional 


requirements (FRD)-Q2 

< 

R4 - Incorrect design 
functional it y-Q2 

< 

R5 - No regression testing - 
T5, M4 

( 

R6 - S/W builds not 


converging to an acceptable 
product - T5, M2 
R7 - Inputs to S/W could 
violate boundary conditions, 
trigger non- tested paths, etc 

- T5, Q2 

R8 - Poor Workmanship in 
the software product 
(spaghetti code, un- 
maintainable code, etc.) - 
QI.Q3 

R9 - Latent S/W defects 
could cause the system to 
fail or not meet it’s 
requirements- T5, Q2, Q5 
RIO - Late awareness (or 
lack of anticipation) of 
schedule, performance, cost 
and quality problems - T5, 
Q5, M2, M3 
R 1 1 - Software safety 
problem - A2, AJ 
R12 - Executing faulty 
commands on a spacecraft - 
Ql, 02 

R 13 - Lack of robustness of 
functions supported by S/W 

- Q3, Q5, A4 

RI4 - S/W fails in a harmftjl 
manner - Al, A2 


( Schedule Pressure 

Resolved by S 
( Repeat Testing 

( Changing Requirements 

< S/W Faults Could Impact 

System Testing/ Schedule 


( Late Problem 

Identification 

< Repair and Repeat Testing 

< Changing Requirements 

< S/W Faults Could Impact 
System Testi 


Negligible or small 


Negligible or small 


Medium/ High Risk 


Program Content 


Tl -Accept Test (w/ 
Metrics i Key Critical 
Functions) 

T2-Functional Test (w / 
Metrics & Key Critical 
Functions) 

T3 -Subsystem integration 
Test 

T 4-Unit Test (basic SW 
Dev Folders) 

T5-FormaI Test Plan 


Analysis 

( A 1 -Hazards Analysis 

< A2-S/W FMEA (critical 

functions) 


Ql -Conformance to S/W 
Standards & Guidelines 
Q2- Requirements Trace 
Q3 -Basic Technical Status 
Reviews (TSRs) including 
critical design and select 
code 

Q4-Light V&V role 
(report to Proj Mgr ) 
Q5-Requirements Mgt. 
(local config mgt ) 


Ml -Minimal S/W QA Plan 
(WPA only) 
M2-Configuration 
Management (Code & 
version control) 

M3 -Milestone Reviews 
(CDR, PDR, ) 


01 -Support Contractor 
Mgt (Assessment of 
critical areas) 

Residual Risks 

Rt- Lack of confidence in 
acceptability of S/W to 
meet system’ s needs-TJ 
R7 • Inputs to S/W could 
violate boundary 
conditions, trigger norv 
tested paths, etc - T5, Q2 
R8 - Poor Workmanship in 
the software product 
(spaghetti code, un- 
maintainable code, etc ) - 
QI.Q3 

R9 - Latent S/W defects 
could cause the system to 
fail or not meet it s 
requirements- T5, Q2, Q5 
RIO - Late awareness (or 
lack of anticipation) of 


Mitigition, Content determined by user 



Program Content 


Residual Risks 


Appropriate Subset of 
Residual Risk Issue 
Relating to Selected 
Program Content 


Schedule Pressure Resolved by 

S 

Repeat Testing 
Changing Requirements 
S/W Faults Could Impact 
System Testi 


Late Problem Identification 
Repair and Repeat Testing 
Changing Requirements 
S/W Faults Could Impact 
System Testing 


Negligible or small 



As Selected (Tailored to 
be Project Specific) 


Program Content 


( Tl -Accept Test (w/ Metrics, 

good functional coverage, & 
witnessing) 

< T2-Full Functional Test (w/ 
Metrics) 

( T3-Subsystem integration Test 
(Metrics) 

< T4-Unit Test (full SW Dev 
Folders) 

( T5-FormaI Test Plan 

Analysis 

< Al -Hazards Analysis 

( A2-S/W FMEA 

( A3-Safety Analysis (critical 

issues) 

( A4-Code Analysis (of critical 

w/automated support) 


Q I -Conformance to S/W 
Standards & Guidelines (QA 
check/peer audit) 
Q2-Requirements Trace 
Q3 -Defined Peer Reviews used 
for TSRs 

Q4-R eportmg to Center 
Director ) 

Q$-Requi remenu Mgt (trace 
CM, CCB) 

Q6-Operations Software QA & 
V&V (critical functions updates 
only) 


< Mi-Full S/W QA Plan 

( M2 -Con figuration Management 

(Code & Version control) 

{ M3-Milestone Reviews (CDR, 

PDR, etc ) 

( M4-Risk Management program 

(basic) 

< M6- Project S/W Metres 
program ( System/ Acc P/FRs) 

Other 

( O I -Support Contractor Mgt. 

(continuous assessment) 

( 02-Misa*on Operations and 

Command Assurance (MOCA) 

Residual Risks 

( R7 - Inputs to S/W could violate 

boundary conditions, trigger 
non-tested paths, etc - T5, Q2 

( R9 - Latent S/W defects could 

cause the system to fail or not 
meet it's requirements- T5, Q2, 

Q5 

( RI I - Software safety problem - 
A2. A3 

< R 12 - Executing faulty 
commands on i spacecraft - Q I . 
02 

< R 1 3 - Lack of robustness of 



Program Contest 


T I -Accept Test (w/ 
Metres, full ftmctiooal 
coverage, & 
witnessing) 

T2-Full Functional Test 
(w f Metrics) 

T3 -Subsystem 
irrtegntion Test 
(Metrics / trend 
tnslysis) 

T 4-Unit Test full SW 
Dev Folders) 

T5-FormaJ Test Plan 


A 1 -Hazards Analysis 
A2-S/W FMEA 
A3- Safety Analysis 
(Full) 

A4-Code Analysis 
(Full) 

A5-S/W Fault Tree 
Analysis 


Ql -Conformance to 
S/W Standards & 
Guidelines (QA critical 
item audit) 

Q2-Requirements Trace 
(complete) 

Q3-S/W 

lnspections(NASA) 
used for TSRs (m 
increased coverage) 
Q4-IV&V 
w independent 
reporting to NASA HQ 
Q 5 -Requirements Mgt 
(trace CM, CCB, tool. 

volatility tracking) 
Q6-Operations 
Software QA & V&V 
(incremental updates) 


Ml -Full S/W QA Plan 
M2 -Configuration 
Management (Full 
coverage w/ mtmdalory 
use of a tool) 
M3-Milestone Reviews 
(CDR, PDR, etc with 
participation of 
independent reviewers 
mandatory) 

M4- Project Risk 
Management program 
MS- Integrated Support 
of Fault Protection 
and/or Failure 
Detection, Isolation & 
Recovery subsystems 
M6-Full Project 
Software Metrics 
program 
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Risk Balance Profile 

Software Quality and V&V Program Guide 
“FODORS” 


Program 

Minimal QA/VAV Procram (Very High Risk) 

Tailored Approach 

■Mmffiiin 


Program Content 

Program Content 

Program Content 

Program 

Tearing 


Testing 

Content 

T1 -Accept Test (pass/fail w/o metrics) 
T2- Functional Test (pass/fail) 

Analysis 

A 1 -Hazards Analysis 


T1 -Accept Test (w/ Metrics, full functional coverage, 
& witnessing) 

T2-Full Functional Test (w/ Metrics) 

T3 -Subsystem integration Test (Metrics / trend 

R 

A2-SAV FMEA (if applicable for critical functions only) 


analysis) 

E 

QA 


T4-Unit Test full SW Dev Folders) 

S 

I 

None 

Related Management 

As Selected (Tailored to 

T5-Formal Test Plan 
Analysis 

D 

None 

be Project Specific) 

A 1 -Hazards Analysis 

U 

A 

Other 

None 


A2-S/W FMEA 
A3-Safety Analysis (Full) 

L 

Residual Risks (Missing Content) 

1- Lack of confidence in acceptability of S/W to meet system’s needs-Tl + 


A4-Code Analysis (Full) 
A5-S/W Fault Tree Analysis 

P 

2 * Unknown functional and system margins-T2+ 


QA 

E 

3 - Inconsistent S/W requirements with respect to the system’s functional 


Q l -Conformance to S/W Standards & Guidelines 

R 

requirements (FRD>Q2 


(QA critical item audit) 

F 

4 - Incorrect design functionality-Q2 


Q2-Requirements Trace (complete) 

O 

5 - No regression testing -T5, M 4 


Q3-S/W Inspections(NASA) used for TSRs (w/ 

R 

6 - S/W builds not converging to an acceptable product - T5, M2 


increased coverage). 

M 

7 - Inputs to S/ W could violate boundary conditions, trigger non-tested 


Q4-IV&V w/independent report mg to NASA HQ 

A 

paths, etc - T5,Q2 


Q5-Requirements Mgt. (trace CM, CCB,. tool. 

N 

8 - Poor Workmanship in the software product (spaghetti code, on- 


volatility tracking) 

C 

maintainable code, etc.) - Ql, Q3 


Q 6 - Opera ti o ns Software QA & V&V (incremental 

E 

9 - Latent S/W defects could cause the system to fail or not meet it’s 
requirements- T5, Q2, Q5 


updates) 

Related Management 

R 

10 - Late awareness (or lack of anticipation) of schedule, performance, cost 


Ml -Full S/W QA Plan 

I 

and quality problems - T5, Q5, M2, M3 


M2 -Configuration Management (Full coverage w> 

S 

1 1 - Software safety problem - A2+, A3 


mandatory use of a tool) 

K 

12 - Executing faulty commands on a spacecraft - Ql, 02 

13 - Lack of robustness of functions supported by S/W - Q3, Q5, A4 

14 - S/W fails in a harmful manner - AI+, A2+ 

1 5 - H/W and system failures compounded by inappropriate S/W responses - 
Q5,M4 

16 - Missing, wrong or extra software requirements -Q 2 , Q3, Q5, M2 


M3-Milestone Reviews (CDR, PDR, etc with 
participation of independent reviewers mandatory) 
M4 -Project Risk Management program 
M5-Integrated Support of Fault Protection and/or 
Failure Detection, Isolation & Recovery subsystems 
M 6 -F 11 II Project Software Metrics program 


17 « Working with out of date requirements - Q2, Q3, Q5, M2 

1 8 - Failure to identify critical QA and V&V processes for S/W - Tl+, T2+, 
A2+ 

19 - Failure to identify critical contractor monitor points - M2, M3, 01 

20 - Can’t identify changes impacts (cost, schedule, functionality, etc.) - M2 

21 - Project progressing to the next phase of development before it is ready - 
Ml, M3 

Residual Risks 

Other 

01 - Support Contractor Mgt. (continuous assessment 
w/ RFP & SEB support from QA and IV&F roles) 

02- Mission Operations and Command Assurance 
(MOCA) 

Residual Risks (Mtssisg Coates') 

28 - Encountering a S/W error that wasn't tested (i.e.. 


22 - Non-standard documentation and source code - Ml, M2 

Appropriate Subset of 

can’t test everything in a complex software product) - 


23 - Unable to effectively add personnel to an “in progress” project - T5, 

Residual Risk Issue 

M1+, M4+ 


Ml 

Relating to Selected 

29 - Uploading faulty software to a spacecraft after 


24 - Unable to make enhancements and changes to the S/W - Ql, Q2, M2 

25 - Un-reusable S/W products - Q2 

26 - Choosing the wrong/high risk contractor to develop software - M4, Ol 

27 - Receiving wrong RFP responses with respect to S/W - M 1 , M2 

28 - Encountering a S/W error that wasn’t tested - Ml, M4 

29 - Uploading faulty software to a spacecraft after launch - T5, M2, 02 

V22FODORS.DOC 

Program Content 

launch - T5+, M2+, 02+ 
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O z: 


Mitigation (Risk Reduction) 

Mitigations 

Mitigations (Risk Redaction) 

1 - Use of an Automatic Code Generator(S,22,24) 

2 - Reusing high quality proven software products (req., design, code, and/or 
test casesX 1, 7,8,9, 13,22,25) 

3 - Using Rapid Prototyping aspects of the software system(l,3,6,l6) 

4 - Simulation of software subsystem(U,lG,l2,I6, 24,29) 

5 - Embedding Assertions in the code (1,3,14,16,17) 

6 - Lessons Ieamed(l,5,t0,18,19,20,2l,22,26) 

7 - Apply PACTS to critical fimctions( 1,3,6,9,10,29) 

8 - Identify critical fonctions( 1,3, 10, 13, 16, 17,2 1,27,28) 

9 - Establish volatility metrics( 1,5, 9,2 1,24,28) 

10 - Use Complexity metrics( 1 ,4,7,8,28) 

1 1 - Early training(8,9, 16,22,23,24,29) 

12 - Cross training^, 9, 16,22,23,24,29) 

13 - Do regression testing( 1,3, 5, 9, 14,24.28) 

14 - Incentivize contractor^, 8,9, 10,2 1,24,26) 

15 - Establish reuse requirements^, 6, 8,9, 13,20, 22,24, ,25,28) 

16 - Use TSRs (inci auto code genXM.8,10,16,17,20,21,22,25,28) 

17 - Insight review of contractor SEI level(8, 10,1 9,2 1,22,26) 

18 - Use EVA metrics( 10,20 ,26) 

19 - Standard documentation formats, repocts(6,8,l 0,1 6,20,2 1,22,23 ,24 ,25) 

20 - Validation of auto code generator(5,6,7,8,9) 

21 - Augmenting V&V with Formal Methods techniques (1,3,14,16,17, 28) 


2 - Reusing high quality proven software products 
(req , design, code, and/or test cases) (*) 

6 • Lessons learned (•) 

7 - Apply PACTS to critical functions (29) 

1 2 - Cross training (29) 

14 - Incentivize contractor (•) 

1 5 - Establish reuse requirements (28) 

21 - Augmenting traditional V&V with Formal 
Methods techniques (formal specification, model 
checking, animating specifications, and/or proofs )) 
(28, •) 

Note: • indicates s general risk reduction 


Note: + indicates that adding stronger content techniques of type could reduce this risk 
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